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ABSTRACT 

Modern  cockpit  instruments  provide  pilots  with  rich  visual  and  auditory  information  about  the  aircraft 
and  the  mission.  However,  the  information  is  not  necessarily  presented  in  the  most  effective  way.  For 
example,  three-dimensional  signals  are  presented  on  two-dimensional  displays.  Moreover,  the  information 
is  often  presented  in  an  abstract  manner,  which  consequently  contributes  to  the  pilot ’s  cognitive  workload. 
Although  the  cockpit  environment  is  standardized  to  meet  general  regulations  and  efficiency,  we 
hypothesize  that  there  is  still  room  for  improvement  since  each  pilot  in  principle  perceives  and  processes 
information  differently,  based  on  individual  constitution,  experience  and  preferences.  Although  pilots  are 
highly  skilled  in  interpreting  present-day  displays,  we  believe  that  their  performance  may  be  further 
enhanced  by  intuitive  displays,  and  also  by  the  possibility  to  customize  the  displays  to  the  individual 
needs.  Method:  We  carried  out  a  simulator  study  in  the  networked  Mission  Simulation  Center  at  TNO. 
Seven  (ex-)operational  F-16  pilots  evaluated  the  functionality  of  a  set  of  new  and  intuitive  displays  in 
realistic  scenario ’s.  The  displays  involved  a  3D  audio  system  (manufactured  by  Terma),  a  tactile  display, 
and  a  dual  layer  display.  The  TNO -built  tactile  display  consisted  of  60  small  vibrating  elements  (factors) 
arranged  around  the  pilot’s  torso  in  five  horizontal  rings  of  twelve  factors  at  30  deg  inten’als.  The  TNO 
dual  layer  display  was  implemented  into  the  right  Multi-Function  Display  by  adding  a  transparent  LCD 
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Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 
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14.  ABSTRACT 

Modern  cockpit  instruments  provide  pilots  with  rich  visual  and  auditory  information  about  the  aircraft 
and  the  mission.  However,  the  information  is  not  necessarily  presented  in  the  most  effective  way.  For 
example,  three-dimensional  signals  are  presented  on  two-dimensional  displays.  Moreover,  the  information 
is  often  presented  in  an  abstract  manner,  which  consequently  contributes  to  the  pilots  cognitive  workload. 
Although  the  cockpit  environment  is  standardized  to  meet  general  regulations  and  efficiency,  we 
hypothesize  that  there  is  still  room  for  improvement  since  each  pilot  in  principle  perceives  and  processes 
information  differently,  based  on  individual  constitution,  experience  and  preferences.  Although  pilots  are 
highly  skilled  in  interpreting  present-day  displays,  we  believe  that  their  performance  may  be  further 
enhanced  by  intuitive  displays,  and  also  by  the  possibility  to  customize  the  displays  to  the  individual  needs. 
Method:  We  carried  out  a  simulator  study  in  the  networked  Mission  Simulation  Center  at  TNO.  Seven 
(ex-)operational  F-16  pilots  evaluated  the  functionality  of  a  set  of  new  and  intuitive  displays  in  realistic 
scenarios.  The  displays  involved  a  3D  audio  system  (manufactured  by  Terma),  a  tactile  display,  and  a  dual 
layer  display.  The  TNO-built  tactile  display  consisted  of  60  small  vibrating  elements  (tactors)  arranged 
around  the  pilots  torso  in  five  horizontal  rings  of  twelve  tactors  at  30  deg  intervals.  The  TNO 
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screen  a  few  centimeters  in  front  of  the  existing  screen,  so  that  information  could  be  presented  in  two 
depth  planes.  The  new  displays  were  controlled  by  direct  voice  input.  The  pilots  could  choose  which  of  the 
following  functions  were  presented  on  either  the  3D  audio  and  the  tactile  display:  Threat  warning 
(derived  from  Radar  Warning  Receiver),  wingman  position  (derived  from  Link- 16),  and  the  Minimum 
Recovery  Altitude  during  weapon  delivery.  The  Dual  Layer  display  always  presented  an  overview  of  the 
current  situation.  After  two  practice  trials,  pilots  tested  the  displays  in  three  operational  simulator 
scenarios  of  20-30  minutes.  Scenarios  were  flown  in  a  two-ship  with  the  wingman  flying  in  another  F-16 
simulator  that  was  networked  with  the  test  pilot’s  F-16  simulator.  All  three  scenarios  had  the  same  level  of 
complexity,  and  contained  several  air  and  ground  threats  that  had  to  be  negotiated  before  a  final  air-to- 
ground  delivery.  Conclusions:  The  pilots  were  highly  enthusiastic  about  both  the  new  displays  and  the 
possibility  to  adjust  the  settings  to  their  personal  preference.  They  differed  in  the  way  they  used  the  new 
displays.  Some  of  them  preferred  one  setting  systematically  throughout  the  whole  scenario,  e.g.  air  threats 
presented  on  the  tactile  display.  Other  pilots  changed  more  frequently  between  different  functions, 
depending  on  the  phase  of  the  scenario.  Results  based  on  self  ratings  and  expert  pilot  observer  data 
indicated  that  pilot  performance  may  be  improved  by  using  more  intuitive  displays  and  a  customized 
cockpit.  Further  research  will  include  improving  the  intuitiveness  and  customization  of  the  cockpit  by  use 
of  Smart  Systems  and  Artificial  Intelligence. 


1.0  INTRODUCTION 

Modem  fighter  aircraft  are  equipped  with  highly  sophisticated  cockpits.  Their  design  is  the  result  of  years 
of  development  and  optimization.  The  cockpit  informs  the  pilot  about  the  status  of  the  aircraft  and  the 
mission,  and  allows  for  the  control  over  the  many  aircraft  systems.  Pilots  are  trained  to  make  effective  use 
of  this.  Because  they  all  operate  the  same  cockpit,  all  pilots  are  trained  in  the  same  way.  In  general,  there 
is  little  about  a  cockpit  that  the  pilot  can  adjust  to  his  own  needs.  This  basically  means  that  the  pilot  must 
adapt  to  the  standard.  The  question  addressed  here  is  whether  a  standard  cockpit  really  is  the  most 
effective  environment  for  the  non-standard  pilot.  People  differ  in  their  information  processing:  for 
example,  some  are  tuned  to  visual  signals,  others  pick  up  auditory  information  more  easily. 

Within  the  RNLAF/TNO  Research  Program  Pilot  Factors,  the  current  study  was  defined  to  investigate 
whether  we  can  improve  the  performance  of  highly  skilled  pilots  by  1)  offering  them  cockpit  information 
in  an  intuitive  way  by  multiple  sensory  modalities,  and  2)  allowing  them  to  customize  the  displays  to  their 
personal  needs.  For  this  purpose,  one  of  the  four  F-16  cockpits  in  the  Mission  Simulation  Center  (MSC) 
was  equipped  with  multi-modal  interfaces  which  can  be  adjusted  by  the  individual  pilot:  A  tactile  torso 
display;  a  3D  audio  head  set;  and  a  dual  layer  display.  These  new  interfaces  are  additional  to  existing 
instruments.  They  do  not  add  information,  but  offer  an  extra  or  alternative  channel  to  display  certain 
parameters.  The  new  displays  are  designed  to  present  information  in  an  intuitive  way.  For  example,  with 
the  3D  audio  interface  one  can  hear  a  sound  coming  from  a  certain  direction.  In  addition  to  these  new 
displays,  a  speech  interface  allowed  pilots  to  control  the  cockpit  by  means  of  direct  voice  inputs. 

The  long  term  goal  of  the  research  program  is  to  have  the  cockpit  adapt  automatically  to  the  pilot.  The 
next  step  in  that  direction  will  be  to  predefine  the  display  settings  that  a  single  pilot  prefers  for  different 
situations.  The  ultimate  step  will  be  to  include  artificial  intelligence  that  monitors  the  development  of  the 
scenario  and  the  pilot’s  capacity,  and  adjusts  the  cockpit  accordingly. 
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2.0  METHODS 

2.1  Mission  Simulation  Centre 

The  TNO/RNLAF  Mission  Simulation  Centre  (MSC)  contains  four  networked  F-16  simulators.  Three 
fixed-base  simulators  feature  original  F-16  cockpits,  and  differ  in  the  type  of  visual  Out-The-Window 
display  (Dome,  Cylinder,  Helmet  Mounted  Display).  Cockpit  instruments,  throttle  and  stick  are  replaced 
by  simulated  replicas.  The  fourth  simulator  is  built  in  the  motion-base  DESDEMONA  facility.  In  the 
current  study,  only  two  of  the  four  cockpits  were  used:  the  cylinder  and  the  dome  (Figure  1).  The  multi¬ 
modal  interfaces  have  been  implemented  in  the  dome-simulator,  designated  Multi-Modal  Suite  (MMS). 


Figure  1 :  Two  F-16  simulators  in  the  Mission  Simulation  Centre.  The  dome  cockpit  (left)  was 
operated  by  the  subject-pilot.  The  cylinder  simulator  (right)  was  operated  by  the  wingman. 


The  simulators  used  the  advanced  F-16  flight  model  by  Bihrle  Applied  Research  (DSix  version  1.95). 
Scenarios  were  developed  in  the  mission  environment  of  Simigon  Airbook  (version  5.2).  The  interface 
with  the  MMS  was  built  by  TNO  using  C++  plug-ins.  Out-the -window  imagery  was  computed  by  Evans 
&  Sutherland  (EPX50)  image  generator,  and  projected  on  three  (cylinder)  and  eight  (Barco  SEER8 
minidome)  Barco  projectors,  respectively. 

2.1.1  Multi-Modal  Suite  (MMS) 

The  cockpit  of  the  Multi-Modal  Suite  (MMS)  was  equipped  with  four  new  interfaces:  Voice  Interface, 
Tactile  Display,  3D  audio  display,  and  Dual  Layer  Display. 

2.2.2  Voice  interface 

The  voice  interface  consisted  of  software  modules  that  use  speech  as  an  input  or  generate  speech  or  audio 
as  an  output.  The  Direct  Voice  Input  (DVI)  module  embeds  an  Automatic  Speech  Recognizer  (ASR),  the 
Direct  Voice  Output  (DVO)  module  is  based  on  a  Text-To-Speech  (TTS)  system,  the  Audio  Dispatching 
Module  (ADM)  is  used  for  sound  generation.  All  modules  operate  independently  but  are  controlled  by  a 
dialogue  manager  by  means  of  a  network  protocol.  The  DVI,  DVO,  and  ADM  modules  are  described 
below. 
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2.2.2.1  Direct  Voice  Input  (DVI) 

For  voice  input  we  implemented  Loquendo  ASR  version  6.7.  As  voice  recognition  is  not  guaranteed  100% 
correct,  voice  input  was  restricted  to  non-critical  actions/commands.  Feedback  on  voice  commands  was 
either  given  in  the  form  of  a  voice  message  (speech  synthesis),  or  by  displaying  on  the  existing  displays 
(e.g.  UFC,  MFD,  HUD).  For  example,  the  former  was  done  with  (de-)  activation  of  a  display,  whereas  the 
latter  was  done  with  the  setting  of  frequencies. 

The  voice  commands  were  designed  in  such  a  way  that  they  were  intuitive,  consistent,  as  short  as  possible 
and  acoustically  distinct  (e.g.  we  use  “on”/“switch  off’  instead  of  “on”/”off ’).  The  recognition  grammar 
was  designed  in  an  iterative  process  in  which  an  expert  user  was  used  to  provide  feedback.  Also,  the 
recognition  accuracies  of  various  versions  of  the  grammar  have  been  tested.  Recognition  accuracy  was 
improved  by  modelling  pronunciation  variation  (adapting  the  acoustic  models  and  optimizing  the  word 
pronunciations)  and  by  training  speaker-dependent  acoustic  models,  resulting  in  word  accuracies  of  98.5% 
(tested  on  an  independent  test  set  of  48  representative  commands).  Table  1  shows  the  voice  commands 
used  for  the  different  functions  used  in  this  study.  The  rows  are  grouped  around  each  of  the  four 
interfaces,  as  indicated  in  the  first  column:  Voice,  Dual  layer  display,  Tactile  display,  and  3D  audio 
display.  Within  each  interface  the  available  functions  are  indicated  by  the  next  two  columns  (column  2-3). 
Columns  2-4  together  compose  the  voice  command  to  activate,  respectively  deactivate  the  corresponding 
functionality.  For  example,  the  voice  command  to  activate  the  function  of  “active  wingman”  on  the  tactile 
display  is  “Body  active  wingman  on”.  In  this  study,  each  display  could  only  present  one  function  at  the 
same  time. 


Function 

Voice  command 

|  Description 

Comm  1 

Preset 

<1-20> 

Comm  2 

Preset 

<1-20> 

1 

On  /  Switch  Off  /  <00-73> 

2 

On  /  Switch  Off  /  <0000-7777> 

IFF 

Mode 

3 

On  /  Switch  Off  /  <0000-7777> 

4 

On  /  Switch  Off  /  Alpha  /  Bravo 

Charly 

On  /  Switch  Off 

All  modes 

On  /  Switch  Off 

Tacan 

Channel 

<1-1 26> 

X-ray  /  Yankee 

Mode 

Switch  Off  /  Receive  (only)  /  Transmit  Receive  /  Air-to-air 

Voice  Commands 

ALOW 

AGL 

MSL 

Bingo 

<0-10.000> 

Bullseye 

<1-29> 

On  /  Switch  Off 

Steerpoint 

<1-1 27> 

Release 

Input  /  Store 

Min  Release 

<0-50. 000> 

Hard  deck 

note:  Hard  deck  is  connected  to  Advisory  Altitude 

Output  /  Say 

Altitudes 

note:  Repeats  Release,  Min  Release  and  Hard  deck  altitudes 

Undo 

Bottom  Layer 

Less  relevant  info 

All  exept  what  is  displayed  on  the  top  layer 

Dual  Layer  Display 

Top  Layer 

Most  relevant  info 

Radar  contacts  and  related  symbology 

Link-16  symbology 

Passive  wingman 

Direction  towards  wingman  (space  oriented)  continious 

Warning  when  wingman  is  close 

Active  wingman 

Direction  towards  wingman  (space  oriented)  on  radio  call 

Tactile  Display 

Body 

Ground  threat 

Direction  towards  ground  threat  (ground  oriented) 

Air  threat 

Direction  towards  air  threat  (ground  oriented) 

Advisory  altitude 

Warning  when  reaching  hard  deck  from  above 

AMRAAM 

Warning  when  missile  goes  active 

On  /  Switch  Off 

All  Body  function  off 

Passive  wingman 

Direction  towards  wingman  (space  oriented)  continious 

Warning  when  wingman  is  close 

Active  wingman 

Radio  call  wingman  from  wingman  direction  (space  oriented) 

(3-D)  Audio  Display 

Audio 

Ground  threat 

Direction  towards  ground  threat  (ground  oriented) 

Air  threat 

Direction  towards  air  threat  (ground  oriented) 

Advisory  altitude 

Warning  when  reaching  hard  deck  from  above 

AMRAAM 

Warning  when  missile  goes  active 

All  Audio  functions  off 

Table  1 :  Voice  commands  and  corresponding  functions. 
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2.2.22  Direct  Voice  Output  (DVO) 

Voice  output  was  used  to  give  feedback  on  the  voice  input  when  speech  is  more  appropriate  than  audio 
signal,  or  when  visual  feedback  was  not  possible.  This  was  for  instance  the  case  for  switching  on/off  new 
interfaces  (e.g.  the  tactile  interface).  For  voice  output,  we  used  Nuance  RealSpeak  Solo  4.0,  USA  male 
voice  (Tom)  Text-to-Speech  software.  Pronunciation  and  speech  tempo  was  tuned  manually. 

2.2.3  Tactile  display 

The  tactile  display  consisted  of  a  vest  with  60  vibrating  elements,  or  so-called  “tactors”,  worn  around  the 
pilot’s  torso,  as  tested  before  in  a  fighter  environment  [1],  The  vest  is  shown  in  Figure  3  (left).  The  tactors 
were  made  of  pager  motors  embedded  in  small  plastic  boxes  (Figure  3,  right).  The  activation  of  the  tactors 
was  computer  controlled.  When  active,  tactors  vibrated  at  a  fixed  frequency  of  150Hz.  The  vibration 
amplitude  was  not  adjustable.  All  tactors  were  arranged  in  five  rings  and  12  columns.  Hence,  signals  could 
be  presented  in  12  segments  of  30  deg  in  five  horizontal  planes.  In  the  sections  below,  the  five  horizontal 
rings  will  be  numbered  1  through  5,  starting  at  the  top. 


Figure  3:  TNO  Tactile  Torso  Display  (left)  and  inside  view  of  factor  (right). 


As  shown  in  Table  1,  the  pilot  could  select  one  of  six  different  functions  on  the  tactile  display: 

2. 2. 3.1.  Passive  wingman  (PW) 

In  this  mode,  the  tactile  display  continuously  indicated  the  3D  direction  in  which  the  wingman  is  located 
relative  to  the  own  ship.  Obviously,  the  limited  spatial  resolution  of  the  tactor  configuration  did  not  allow 
for  a  1:1  mapping  of  all  relative  directions.  For  that  reason,  the  horizontal  direction  (azimuth  angle)  was 
projected  onto  the  tactor  being  closest.  The  vertical  indication  was  coded  in  five  segments.  The  middle 
tactor  ring  (3)  was  used  when  the  vertical  position  of  the  wingman  was  within  -15  and  +15  deg  relative  to 
the  own  level.  Angles  between  +15  and  +45  deg  (-15  and  -45  deg)  were  projected  on  tactor  ring  2 
(respectively  4).  Angles  between  +45  and  +90  (-45  and  -90)  were  projected  on  the  upper  (1)  or  lower 
tactor  ring  (5).  The  direction  of  the  wingman  was  corrected  for  attitude  changes  of  the  own  ship,  so  that 
the  wingman  was  always  presented  in  egocentric  coordinates.  Tactors  were  activated  with  a  200ms 
on/200ms  off  rhythm. 
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2.2. 3.2  Active  wingman  (AW) 

The  relative  position  of  the  wingman  was  similar  as  in  the  passive  wingman  mode,  but  only  when  the 
wingman  was  communicating  (this  mode  was  linked  to  a  push-to-talk  button  of  the  wingman). 

2.2. 3. 3.  Ground  threat  (GT) 

The  azimuth  angle  of  ground  threat  was  indicated  on  the  12  tactors  of  ring  5.  In  inverted  flight  (aircraft 
attitude  >  90°)  the  direction  was  mirrored  to  ring  1,  so  as  to  maintain  a  rough  egocentric  coordinate 
system.  A  maximum  of  two  threats  could  be  presented  at  the  same  time.  The  duty  cycle  for  both  stimuli 
was  1400ms.  The  tactor  representing  the  first  threat  was  activated  three  times  at  the  beginning  of  a  duty 
cycle  with  100ms  on/ 100ms  off.  The  second  threat  was  indicated  in  anti -phase  with  the  first  threat  signal 
(Figure  4). 


Duty  cycle  1400ms 


Threat  1 


100  ms 


Threat  2 


Figure  4:  Activation  pattern  of  two  simultaneous  tactile  warnings. 


2.2. 3.4.  Air  threat  (AT) 

In  the  air  threat  mode,  the  tactile  display  presented  the  direction  of  a  maximum  of  two  enemy  aircraft  with 
the  same  activation  pattern  as  the  ground  threat  mode  (Figure  4).  However,  where  the  ground  threat  mode 
only  indicated  the  2D  horizontal  direction,  the  air  threat  mode  also  presented  the  3D  vertical  direction 
similar  to  the  wingman  mode.  The  activation  pattern  was  100ms  on/lOOms  off. 

2.2. 3.5.  Advisory  Altitude  (AA) 

This  function  was  used  in  air-to-ground  situations,  and  comprised  two  signals.  The  first  signal  was 
presented  on  three  consecutive  tactors  mid-front  ring  2,  which  alerted  the  pilot  about  approaching  the 
minimum  release  altitude  and  was  a  function  of  the  dive  angle.  The  signal  consisted  of  one  burst  of  four 
activations  with  200ms  on/200ms  off.  The  second  signal  alerted  the  pilot  about  crossing  hard  deck. 
Because  of  its  importance,  this  signal  was  made  extra  strong:  During  3  seconds,  all  12  tactors  of  ring  4 
were  activated  200ms  on/200ms  off. 

2. 2. 3. 6.  AMRAAM 

In  this  mode,  a  single  tactor  was  activated  briefly  when  the  AMRAAM  became  active. 
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2.2.4  3D  audio  interface 

The  same  functions  described  for  the  tactile  display  could  be  presented  on  the  Terma  3D  audio  system 
(Denmark).  Head  tracking  was  performed  by  a  Polhemus  Fastrak  RevE  headtracker.  For  all  pilots  we  used 
the  default  Head-Related  Transfer  Function  (HRTF’s),  as  was  provided  with  the  system. 

2.2.5  Dual  Layer  display 

The  TNO  “Dual -Layer  Display”  (DLD)  provides  one  extra  depth  layer  over  standard  (flat)  displays, 
increasing  the  information  capacity  of  the  display  (Kooi,  in  print).  The  DLD  consists  of  two  orthogonally 
placed  LCD  displays,  which  are  viewed  by  the  observer  via  a  semi  transparent  mirror  placed  with  an  angle 
of  45  deg.  In  this  way  information  of  one  of  the  displays  is  presented  in  the  foreground  relative  to  the  other 
display  which  is  seen  in  the  background.  The  distance  was  set  at  12mm,  which  is  the  maximum  value  for 
the  prototype  DLD  used  here  (with  LCD  displays  with  a  diameter  of  6.4  inch).  In  the  MMS  the  DLD  was 
built  in  as  the  right  Multi  Lunctional  Display  (MED)  for  presenting  information  of  the  radar  and  Link- 16 
(Ligure  5).  At  the  time  of  the  study  the  DLD  could  not  be  adjusted  real-time,  and  throughout  the 
experiment  it  presented  an  overview  of  the  situation.  Because  pilots  could  not  adjust  the  DLD,  we  will  not 
discuss  the  results  of  the  DLD. 


Figure  5:  Dual  Layer  Display  implemented  in  the  MFD.  The  left  and  right  panel  shows  the  rear 
and  front  display,  respectively,  and  the  right  panel  shows  the  two  layers  on  top  of  each  other. 


2.3  Operational  scenarios 

2.3.1  Scenario  description 

The  operational  scenarios  were  as  close  as  possible  to  real  training  scenarios.  The  mission  was  to  fly  as  a 
2-ship  through  a  defended  area  and  attack  a  ground  target.  The  pilots  received  an  “air  picture”  via  the 
Link- 16  network  but  did  not  receive  any  fighter  control.  Due  to  limitations  of  the  MSC  and  the  MMS,  the 
scenarios  were  divided  into  three  main  phases:  the  Ground-to-Air  phase,  the  Air-to-Air  phase  and  the  Air- 
to-Ground  phase.  Figure  6  gives  a  schematic  set-up  of  an  Air-to-Air  phase  in  one  of  the  scenarios.  During 
the  Ground-to-Air  phase  the  formation  encountered  two  SAM-sites,  during  the  Air-to-Air  phase  up-to  six 
hostile  aircraft  before  they  reached  the  target  area.  During  the  Air-to-Ground  phase  the  target  attack  took 
place.  In  total  there  were  six  scenarios,  which  had  the  same  ingredients  but  different  initial  conditions.  The 
scenarios  differed  by  changing  the  presentation  of  hostile  aircraft.  Every  presentation  forced  the  subject 
pilot  to  make  a  different  decisions.  The  scenarios  had  the  same  difficulty  level.  This  way,  each  pilot 
carried  out  six  different,  but  comparable  scenarios. 
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Figure  6:  Example  of  Air-to-Air  phase,  showing  three  hostile  groups  and  one  friendly.  Viperl  is 

the  own  ship  formation. 


2.3.2  Scenario  difficulty  level 

The  limitations  of  the  MSC  limited  the  difficulty  level  of  the  scenario’s.  Normally  the  above  mentioned 
phases  could  happen  at  the  same  time.  The  MMS,  in  it’s  current  version,  did  not  allow  multiple  threat 
categories  to  be  displayed  at  the  same  time.  For  this  reason  the  threat  phases  were  separated  otherwise  the 
pilots  would  have  to  make  an  unrealistic  choice  based  on  expected  threat  priority.  The  impact  on  the 
experiment  would  be  unpredictable.  Also  the  computer  generated  threats  were  not  “smart”  enough  to 
oppose  a  realistic  difficulty  level.  For  this  reason  the  pilots  had  a  limited  weapon  load  against  multiple 
threats  which  increased  the  difficulty  level.  The  expert  pilot  acting  as  wingman  also  increased  the 
difficulty  level  by  forcing  the  subject  pilot  to  divide  his  attention  between  threat  and  formation  integrity. 
Although  the  scenario  difficulty  was  acceptable  for  the  experiment  they  were  expected  to  be  insufficient  to 
increase  the  pilot  workload  to  a  level  which  would  allow  produce  noticeable  performance  differences. 

2.4  Procedure 

The  study  consisted  of  three  sessions,  separated  over  three  different  days.  The  first  session  was  to 
familiarize  the  pilots  with  the  MMS.  The  control  of  the  various  displays  and  their  functionality  was 
explained  and  practiced.  In  the  second  session,  the  pilots  were  given  the  opportunity  to  determine  their 
personal  preferences.  In  the  third  and  final  session,  the  MMS  was  tested  in  the  operational  scenarios,  and 
compared  with  the  standard  F-16  cockpit.  In  the  latter  session  the  actual  experiment  took  place.  Pilots 
carried  out  the  six  operational  scenarios  described  above.  A  block  of  three  of  the  scenarios  was  flown  with 
the  use  of  the  MMS.  The  subject  pilot  would  start  with  his  personal  settings  as  determined  in  the  second 
session,  although  he  was  free  to  deviate  from  this  whenever  he  felt  the  need.  A  second  block  of  three 
scenarios  was  flown  in  the  traditional  cockpit.  The  same  order  of  six  scenarios  was  used  for  all  pilots, 
because  all  scenarios  were  comparable  (containing  the  same  phases  and  type  of  events).  The  order  of  the 
two  blocks  (with  and  without  MMS)  was  counterbalanced  between  pilots.  The  subject  pilot  flew  as 
formation  lead  while  the  expert  pilot  flew  as  wingman. 
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2.5  Measurements 

Directly  after  each  scenario,  a  10  minute  debriefing  was  performed  by  the  expert  pilot,  asking  the  subject 
pilot  to  recall: 

•  The  flow  or  “spaghetti”  (i.e.,  course  of  most  important  moments,  actions  and  results  in  order  of 
occurrence) 

•  Situation  awareness  related  to  threat  and  wingman 

•  Comments  on  the  displays  and  functionalities 

The  MSC  operator  and  expert  pilot  noted  all  voice  inputs  made  by  the  pilot,  so  as  to  examine  the  exact  use 
of  the  MMS  during  each  phase  of  the  scenario.  In  addition,  the  pilots’  opinion  was  collected  using  two 
different  questionnaires: 

•  Questionnaire  1  -  Functionality  of  the  MMS  in  the  operational  test  scenarios 

•  Questionnaire  2  -  Expected  usefulness  of  MMS  in  actual  operations 

Questionnaire  1  contained  a  total  of  44  statements  about  the  ergonomics  and  functionality  of  the  MMS 
interfaces,  which  could  be  answered  on  a  seven-point  scale  (ranging  from  “totally  disagree”  to  “totally 
agree”).  There  were  12  statements  about  the  tactile  display,  and  the  same  12  statements  about  the  3D  audio 
display.  Five  statements  concerned  the  DLD  as  well  as  the  Voice  Interface,  and  ten  more  statements  dealt 
with  the  MMS  concept  in  general.  The  questionnaire  was  filled  out  at  the  end  of  third  (experimental) 
session  at  the  end  of  the  study. 

Questionnaire  2  was  used  to  determine  how  the  pilots’  expectations  and  opinion  about  the  MMS 
developed  throughout  the  study.  It  was  administered  before  the  start  of  the  study,  and  subsequently  after 
each  of  the  three  sessions.  Ratings  were  given  on  a  five -point  scale  (ranging  from  “not  at  all  useful”  to 
“extremely  useful”).  A  five -point  scale  was  chosen  instead  of  the  seven-point  scale,  to  make  the  answers 
better  comparable  to  the  logging  system  used  by  the  expert  pilot. 

2.5.3  Subject  pilots 

Seven  RNLAF  F-16  pilots  participated  as  subject  in  this  experiment.  Two  of  them  were  very  experienced 
retired  F-16  pilots  which  have  been  out  of  service  for  less  than  two  years.  Five  pilots  still  operationally  fly 
within  the  RNLAF.  The  experience  of  all  pilots  ranged  from  1000  to  2300  flying  hours  on  the  F-16  with 
an  operational  status  ranging  from  2-ship  lead,  force-lead  to  weapons  instructor  and  test  pilot. 

2.5.4  Expert  pilot 

The  expert  pilot  was  a  retired  RNLAF  F-16  experimental  test  pilot  and  weapons  instructor.  He  flew  over 
2000  hours  F-16.  During  the  experiment  he  observed  the  subject  pilots  and  flew  as  wingman. 


3.0  RESULTS 
3.2.1  Pilot  performance 

All  subject  pilots  noticed  that  the  MMS  increased  their  threat  awareness,  especially  for  ground  threats. 
Four  pilots  used  the  MMS  to  improve  their  wingman  awareness.  All  pilots  used  the  MMS  for  altitude 
awareness  helping  them  to  prevent  crossing  the  “hard  deck”.  The  MMS  provided  additional  and  intuitive 
information.  One  pilot  said  he  felt  “more  at  rest”  as  the  MMS  lowered  his  task  saturation.  The  increased 
ground  threat  awareness  gave  another  pilot  a  “warm  fuzzy  feeling”  as  he  knew  more  about  the  threat 
positions  and  found  it  easier  to  find  missile  launches.  All  pilots  and  the  expert  pilot  subjectively  noticed  a 
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difference  in  performance  between  flying  with  and  without  the  MMS.  The  increased  awareness  level  with 
the  MMS  allowed  the  pilots  to  make  more  efficient  decisions  and  decreased  their  workload.  Due  to  the 
limited  difficulty  of  the  scenarios  the  subject  pilots  were  not  pushed  to  their  performance  limits.  All 
subject  pilots,  and  the  expert  pilot,  expected  that  the  effect  of  the  increased  awareness  level  with  the  MMS 
would  become  more  prominent  when  flying  more  difficult  scenarios. 

3.2.2  Operational  use  of  MMS 

The  operational  use  of  the  tactile  and  3-D  audio  display  is  shown  in  Figure  7  for  each  individual  pilot. 

The  data  clearly  shows  similarities  and  differences  in  preferred  uses  of  the  MMS  between  subject  pilots. 
While  five  pilots  used  the  tactile  and  audio  displays  to  increase  their  threat  and  wingman  awareness,  two 
pilots  only  used  the  displays  for  threat  awareness.  Both  preferred  using  the  Link- 16  data  for  their  wingman 
awareness.  Most  pilots  switched  between  functions  on  their  displays  between  phases,  but  some  minimised 
switching  and  constantly  kept  the  same  function  active  as  a  way  to  avoid  confusion  about  the  signal.  By 
minimising  switching  they  also  reduced  the  workload  to  manage  the  MMS  and  prevent  forgetting  to 
switch  to  the  intended  function,  which  actually  happened  several  times. 


Figure  7:  Chart  showing  the  selected  functions  on  tactile  and  3D  audio  display  in  different 
phases  of  the  mission.  “OFF”  means  display  OFF.  GT=Ground  Threat  (green);  AT=  Air  Threat 
(blue);  PW=Passive  Wingman  (orange);  AW=Active  Wingman  (yellow);  AA=Advisory  Altitude 

(red). 


3.2.2. 1  Tactile  display  use 

All  pilots  preferred  tactile  ground  threat  indications  during  the  Ground-Air  phase  which  allowed  them  to 
react  to  the  threat  without  abandoning  other  tasks.  Three  pilots  preferred  tactile  air  threat  indications 
during  the  Air-Air  phase  while  two  pilots  preferred  to  continuously  have  tactile  ground  threat  displayed 
during  this  phase.  Both  pilots  wished  to  maintain  clarity  in  the  warnings  on  the  different  displays.  Two 
pilots  maintained  clarity  between  the  displays  by  using  the  tactile  display  for  the  expected  threat  (ground 
or  air)  while  using  the  3-D  audio  display  for  wingman  awareness.  Only  one  pilot  used  the  tactile  display 
for  wingman  awareness  during  the  Air-Air  phase.  Four  pilots  used  the  tactile  display  for  altitude 
awareness  during  the  Air-Ground  phase  while  the  others  maintained  tactile  threat  or  wingman  indications 
during  this  phase.  In  general  the  tactile  display  was  mainly  used  for  threat  indications. 

As  shown  in  Figure  8,  pilots  found  that  two  practice  trials  offered  sufficient  practice  with  the  tactile 
display  (question  1),  indicating  the  intuitive  nature  of  the  display.  All  pilots  were  positive  about  the  clarity 
of  tactile  signals,  and  all  but  pilot  3  found  them  easy  to  interpret  (questions  3-5),  especially  with  respect  to 
the  stimulus  direction  (question  7).  Not  all  pilots  rated  the  comfort  of  the  tactile  display  as  high  (question 
2).  The  prototype  tactors  are  encased  in  a  plastic  box,  which  presses  against  the  pilots’  back.  The 
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resolution  of  the  tactile  display  was  judged  reasonable,  but  not  perfect  (question  6).  Only  two  pilots  could 
easily  discriminate  different  tactile  signals  (questions  9  and  10). 


totally  disagree  totally  agree 


°  1 
D  6 
a  5 
o  2 

+  3 

•  4 

1  7 


Figure  8:  Individual  ratings  of  tactile  display. 


3. 2. 2. 2  Audio  display  use 

Three  pilots  used  3-D  audio  for  wingman  awareness  during  the  Ground-Air  phase  while  two  pilots  used  it 
for  ground  threat  and  two  for  air  threat.  Four  pilots  used  3-D  audio  for  air  threat  awareness  during  the  Air- 
Air  phase  while  two  use  it  for  wingman  awareness.  One  pilot  did  not  use  3-D  audio  at  all  during  the  Air- 
Air  phase.  He  thought  3-D  audio  was  not  that  effective.  Five  pilots  used  the  extra  audio  warning  for 
altitude  awareness  during  the  Air-Ground  phase  while  two  pilots  used  3-D  audio  for  wingman  awareness 
during  this  phase.  In  general,  the  audio  display  was  used  by  half  of  the  pilots  for  threat  awareness  while 
the  others  preferred  wingman  awareness. 

Figure  9  shows  the  results  for  the  3D  audio  display.  Not  all  pilots  found  the  signals  entirely  comfortable. 
Compared  to  the  tactile  display,  3D  audio  signals  seemed  to  be  more  difficult  to  detect  and  interpret 
(questions  3-5).  According  to  question  6,  the  resolution  of  the  audio  display  was  not  perfect.  Since  we 
used  one  generic  transfer  function  to  compute  the  simulated  direction  of  audio  signals,  this  can  be 
improved  by  using  customized  transfer  functions.  Simultaneously  presented  signals  in  the  air  threat  mode 
were  not  discriminated  very  well  (question  9).  Different  signals  for  different  function,  however,  could  be 
discriminated  better  (question  10). 
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Figure  9:  Individual  ratings  of  3D  audio  display. 


3. 2. 3. 5  Judgments  on  MMS  concept 

Figure  10  shows  the  pilots’  judgments  of  the  MMS  concept.  Clearly,  the  ratings  are  quite  scattered.  All 
pilots  were  mildly  to  wildly  positive  about  the  statement  that  the  MMS  increased  the  mission  effectiveness 
(question  1).  Two  out  of  seven  pilots  indicated  to  be  distracted  by  the  MMS  displays  (question  2).  Four 
pilots  felt  that  the  MMS  helped  to  lower  workload;  three  others  disagreed  (question  3).  Five  of  them  were 
able  to  fly  more  heads  up  with  the  MMS  (question  4).  Most  pilots  were  able  to  scan  the  displays  and  found 
them  complementary  (questions  5,  6  and  7).  Without  feedback  on  the  display  status  it  was  difficult  to 
remember  which  MMS  function  was  active  (question  8).  Finally,  the  majority  of  pilots  was  content  with 
their  individual  settings  for  operational  conditions  (questions  9  and  10). 


totally  disagree  totally  agree 
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4.0  CONCLUSIONS 

The  two  objectives  of  this  study  were  to  show  that:  1)  Pilots  use  the  possibility  to  personalize  the  cockpit 
displays  to  their  own  needs;  and  2)  The  MMS  displays  have  added  value  in  operational  scenarios.  Based 
on  the  expert  evaluation  and  the  questionnaires  we  conclude  that  the  study  succeeded  in  both  objectives. 
All  pilots  were  very  positive  about  the  Smartcard  concept  and  the  intuitive  MMS  displays.  During  the 
final  experiment  with  operational  scenarios  they  all  used  MMS  features,  although  they  were  free  to  decide 
not  to  use  any  of  them  and  just  use  the  traditional  cockpit. 

In  line  with  the  first  main  objective,  pilots  clearly  differed  in  their  personal  preference  of  MMS  settings. 
This  shows  that  personalization  of  the  cockpit  (interfaces)  is  desired  and  works.  Concerning  the  second 
objective,  most  pilots  adjusted  their  setting  for  each  phase  in  the  mission,  so  as  to  optimize  the  support 
from  the  MMS  and  increase  their  operational  efficiency  (although  some  pilots  minimized  switching 
between  functions  to  avoid  confusion  about  the  setting,  or  to  prevent  forgetting  to  switch  to  the  intended 
function).  All  subject  pilots  agreed  that  the  Smartcard  concept  increased  their  awareness,  and  as  such 
offers  added  value  in  operational  conditions. 

The  study  revealed  many  other  useful  observations,  and  pilot  comments.  As  we  had  expected,  there  were 
many  suggestions  to  make  necessary  technical  improvements  of  the  MMS.  Most  important,  pilots  were 
missing  adequate  feedback  about  the  status  of  the  MMS  displays,  and  although  they  indicated  to  have 
sufficient  practice,  they  had  some  difficulty  in  managing  the  MMS.  The  MMS  displays  themselves  can 
also  be  improved.  For  example,  voice  input  (recognition)  was  not  error  free  to  the  extent  that  pilots  would 
rely  on  it  in  actual  flight.  Tactile  signals  were  not  yet  optimized  for  the  different  functions,  especially 
where  two  or  more  signals  are  presented  simultaneously.  Not  all  pilots  were  satisfied  about  the  directional 
hearing  in  the  3-D  audio  system,  which  may  be  due  to  the  fact  that  we  used  a  generic  Head-Related 
Transfer  Function. 

Based  on  these  findings,  we  will  further  develop  the  MMS  so  that  multiple  functions  can  be  displayed  on 
the  same  display  at  the  same  time.  This  allows  for  presenting  high-priority  signals  (e.g.  threat)  on  top  of 
another  signal  (e.g.  wingman).  Because  this  feature  was  not  available  in  this  study,  we  had  to  design  the 
operational  scenarios  into  clearly  separated  phases.  With  this  feature,  events  can  be  presented  in  a  more 
random  way,  increasing  the  complexity  of  the  scenario.  Ground  and  air  threat  change  quickly,  and  be  a 
factor  at  the  same  time  and  during  the  ground  attack.  This  could  increase  the  number  of  required  switches 
between  functions  as  priorities  change  quickly.  In  addition,  we  will  upgrade  the  content  and  artificial 
intelligence  of  the  MMS  mission  environment  so  as  to  include  more  smart  components.  This  will  allow  for 
even  more  demanding  scenarios.  Finally,  automatic  or  “smart”  switching  functions  will  be  implemented  to 
help  the  pilot  to  reduce  workload  managing  the  MMS  and  prevent  forgetting  to  switch.  Pilots  will  be  able 
to  program  the  MMS  to  meet  their  personal  preference.  This  “smartcard”  would  control  the  MMS  to 
present  the  pilots  with  the  information  most  suited  for  the  phase  of  the  mission.  Figure  7  illustrates  the 
idea  of  this  smartcard,  being  a  personal  setting  depending  for  different  phases  of  the  mission.  An 
automatic  system  should  be  able  to  detect  each  important  phase  of  the  mission,  and  the  smart  system 
should  be  able  to  predict  the  most  suitable  setting. 
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